<!DOCTYPE html>
<html>
  <head>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <!-- Do we need additional share metadata included here? -->

    <!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=G-Q74F5RJLXB"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'G-Q74F5RJLXB');
</script>

    <title>Exploring a Sentiment Classifier</title>
    
        <link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons">
    <link rel="stylesheet" href="/lit/assets/css/material.min.css">
    <link rel="stylesheet" href="/lit/assets/css/material.min.css">
    <link rel="stylesheet" href="/lit/assets/css/prism-material-light.css">
    <link href="https://fonts.googleapis.com/css?family=Open+Sans:400,400italic,500,500italic,700,700italic" rel="stylesheet" type="text/css">
    <link href="https://fonts.googleapis.com/css?family=Manrope:400,400italic,500,500italic,700,700italic" rel="stylesheet" type="text/css">

    <!-- <link href='/lit/assets/css/main.css' rel='stylesheet' type='text/css'> -->
    <link href='/lit/assets/css/new.css' rel='stylesheet' type='text/css'>
    <link rel="icon" href="/lit/assets/images/favicon.png" type="image/png"/>

  </head>

  <body>
    <div class="mdl-layout mdl-layout--no-desktop-drawer-button mdl-js-layout mdl-layout--fixed-header">
      
      <header class="mdl-layout__header">
        <div class="mdl-layout__header-row">
          <!-- Title -->
          <img class='status-emoji' src="/lit/assets/images/favicon.png"></img>
          <span class="mdl-layout__title">
            <a href="/lit/">
              Learning Interpretability Tool
            </a>
          </span>
          <!-- Add spacer, to align navigation to the right -->
          <div class="mdl-layout-spacer"></div>
          <!-- Navigation. We hide it in small screens. -->
          <nav class="mdl-navigation mdl-layout--large-screen-only">
            <a class="mdl-navigation__link" href="/lit/documentation/getting_started.html">GETTING STARTED</a>
            <a class="mdl-navigation__link" href="/lit/tutorials/">TUTORIALS</a>
            <a class="mdl-navigation__link" href="/lit/demos/">DEMOS</a>
            <a class="mdl-navigation__link" href="/lit/documentation/">DOCUMENTATION</a>
            <a class="mdl-navigation__link" href="/lit/documentation/faq.html">FAQs</a>
            <a class="mdl-navigation__link" href="https://groups.google.com/g/lit-annoucements" target="-_blank">STAY UP TO DATE<img class="header-arrow" src="/lit/assets/images/arrow-link-out.png"/></a>
            <a class="mdl-navigation__link" href="https://github.com/pair-code/lit" target="-_blank">GITHUB<img class="header-arrow" src="/lit/assets/images/arrow-link-out.png"/></a>
          </nav>
        </div>
      </header>
      <div class="mdl-layout__drawer">
        <span class="mdl-layout__title"><a href="/lit/">Learning Interpretability Tool</a></span>
        <nav class="mdl-navigation">
          <a class="mdl-navigation__link" href="/lit/documentation/getting_started.html">GETTING STARTED</a>
          <a class="mdl-navigation__link" href="/lit/tutorials/">TUTORIALS</a>
          <a class="mdl-navigation__link" href="/lit/demos/">DEMOS</a>
          <a class="mdl-navigation__link" href="/lit/documentation/">DOCUMENTATION</a>
          <a class="mdl-navigation__link" href="/lit/documentation/faq.html">FAQs</a>
          <a class="mdl-navigation__link" href="https://groups.google.com/g/lit-annoucements" target="_blank">STAY UP TO DATE<img class="header-arrow" src="/lit/assets/images/arrow-link-out.png"/></a>
          <a class="mdl-navigation__link" href="https://github.com/pair-code/lit" target="-_blank">GITHUB<img class="header-arrow" src="/lit/assets/images/arrow-link-out.png"/></a>
        </nav>
      </div>

      
      <main class="mdl-layout__content hero-banner">
        <div class="tutorial-page-container mdl-grid">
          <div class="mdl-cell--8-col mdl-cell--8-col-tablet mdl-cell--4-col-phone">
            <div class="tutorial-breadcrumbs">
  <a href="/lit/tutorials">Tutorials</a> > <a href="/lit/tutorials/#analysis">Analysis</a> > Sentiment
</div>
            <h2>Exploring a Sentiment Classifier</h2>
<div class="link-out"><a href="../../demos/glue.html" target="_blank">Explore this demo yourself.</a><img class="external-arrow" src="/lit/assets/images/arrow-link-out.png"/></div>
<p>Or, run your own with <a href="https://github.com/PAIR-code/lit/blob/main/lit_nlp/examples/glue/demo.py"><code>examples/glue/demo.py</code></a></p>
<p>How well does a sentiment classifier handle negation? We can use LIT to interactively ask this question and get answers. We loaded up LIT the development set of the Stanford Sentiment Treebank (SST), which contains sentences from movie reviews that have been human-labeled as having a negative sentiment (0), or a positive sentiment (1). For a model, we are using a BERT-based binary classifier that has been trained to classify sentiment.</p>
<p>Using the search function in LIT’s data table, we find the 67 datapoints containing the word “not”. By selecting these datapoints and looking at the Metrics Table, we find that our BERT model gets 91% of these correct, which is slightly higher than the accuracy across the entire dataset.</p>
<div class="mdl-cell mdl-cell--12-col mdl-cell--6-col-tablet mdl-cell--4-col-phone">
  <img class="tutorial-image" src="/lit/assets/images/lit-metrics-not.png"
    onclick="event.target.nextElementSibling.showPopover()" />
  <div class="tutorial-image-popover" popover>
    <img class="tutorial-image" src="/lit/assets/images/lit-metrics-not.png"
      onclick="event.target.parentElement.hidePopover()" />
  </div>
  <div class="tutorial-caption">Above: A comparison of metrics on datapoints containing "not" versus the entire dataset.</div>
</div>
<p>But we might want to know if this is truly robust. We can select individual datapoints and look for explanations. For example, take the negative review, “It’s not the ultimate depression-era gangster movie.”. As shown below, salience maps suggest that “not” and “ultimate” are important to the prediction. We can verify this by creating modified inputs, using LIT’s datapoint editor. Removing “not” gets a strongly positive prediction from “It’s the ultimate depression-era gangster movie.”.</p>
<div class="mdl-cell mdl-cell--12-col mdl-cell--6-col-tablet mdl-cell--4-col-phone">
  <img class="tutorial-image" src="/lit/assets/images/lit-not-saliency.png"
    onclick="event.target.nextElementSibling.showPopover()" />
  <div class="tutorial-image-popover" popover>
    <img class="tutorial-image" src="/lit/assets/images/lit-not-saliency.png"
      onclick="event.target.parentElement.hidePopover()" />
  </div>
  <div class="tutorial-caption">Above: Prediction saliency of the original sentence, including "not".</div>
</div>
<div class="mdl-cell mdl-cell--12-col mdl-cell--6-col-tablet mdl-cell--4-col-phone">
  <img class="tutorial-image" src="/lit/assets/images/lit-saliency.png"
    onclick="event.target.nextElementSibling.showPopover()" />
  <div class="tutorial-image-popover" popover>
    <img class="tutorial-image" src="/lit/assets/images/lit-saliency.png"
      onclick="event.target.parentElement.hidePopover()" />
  </div>
  <div class="tutorial-caption">Above: Prediction saliency of the altered sentence, with "not" removed.</div>
</div>
<p>Using the LIT features of data table searching, the metrics table, salience maps, and manual editing, we’re able to show both in aggregate and in a specific instance, that our model handles negation correctly.</p>

          </div>
          <div class="mdl-cell--4-col hide-me">
            <div class="tutorial-info-container">
              <div class="tutorial-info-header">time to read</div>
              <div class="tutorial-info-copy">3 minutes</div>
              <div class="tutorial-info-header">takeaways</div>
              <div class="tutorial-info-copy">Learn about how the metrics table and saliency maps assisted an analysis of a sentiment classifier's performance when dealing with negation.</div>
            </div>
          </div>
          <div class="tutorial-footer mdl-cell--8-col mdl-cell--8-col-tablet mdl-cell--4-col-phone">
            The Learning Interpretability Tool is being actively developed and documentation is likely to change as we improve the tool. We want to hear from you! Leave us a note, feedback, or suggestion on our <a href="https://github.com/pair-code/lit/issues/new" target="_blank">Github</a>.
          </div>
        </div>

        <div class="footer-container mdl-grid">
  <div class="mdl-cell mdl-cell--2-col mdl-cell--2-col-tablet mdl-cell--4-col-phone"><a href="https://pair.withgoogle.com/" target="_blank"><img src="/lit/assets/images/pair-logo.svg"/></a></div>
  <div class="mdl-cell mdl-cell--2-col mdl-cell--2-col-tablet mdl-cell--4-col-phone">
    <a href="https://research.google/teams/language/" target="_blank">Google Research - Language</a>
  </div>
  <div class="mdl-cell mdl-cell--2-col mdl-cell--2-col-tablet mdl-cell--4-col-phone"><a href="https://github.com/pair-code" target="_blank">Github</a></div>
  <div class="footer-icons mdl-cell mdl-cell--4-col mdl-cell--8-col-tablet mdl-cell--4-col-phone">
    <a href="mailto:peopleai@google.com" target="_blank"><img src="/lit/assets/images/mail.png"/></a>
    <a href="https://medium.com/people-ai-research" target="_blank"><img src="/lit/assets/images/medium.png"/></a>
    <a href="https://www.youtube.com/channel/UCnnns-uu4yy9BXfYSPIX5AA" target="_blank"><img src="/lit/assets/images/youtube.png"/></a>
  </div>
</div>

      </main>

    </div>
  </body>

  

  <script defer src="/lit/assets/js/material.min.js"></script>
  <script defer src="/lit/assets/js/material-components-web.min.js"></script>
</html>
